# • 



Docket No. 003400.P009X 
Express Mail No. EL34499961US 



UNITED STATES PATENT APPLICATION 



FOR 



METHOD OF ITERATIVE SEGMENTATION OF DIGITAL PICTURE 



INVENTORS: 

Dr. Martin Baatz 
Dr. Gerd Binnig 
Dr. Peter Eschenbacher 
Dr. Andreas Melchinger 
Michael Sogtrop 



PREPARED BY: 

BLAKELY SOKOLOFF TAYLOR & ZAFMAN 
12400 Wilshire Blvd., 7th Floor 
Los Angeles, CA 90025-1026 
(310) 207-3800 



003400.P009X Express MaiUJo.: EL634499961US 



METHOD OF ITERATIVE SEGMENTATION OF A DIGITAL PICTURE 

CROSS REFERENCE TO RELATED APPLICATIONS 

[0001] This application is a continuation-in-part of United States Patent 

Application Serial No. 09/023,670, filed February 10, 1998. 

BACKGROUND OF THE INVENTION 

Field of the Invention 

[0002] The present invention relates to a method for segmentation of a 

digital picture consisting of a multiplicity of single picture elements which is in 
particular suitable for digital picture processing and /or object and pattern 
recognition. 

DESCRIPTION OF THE PRIOR ART 

[0003] In most fields of application of digital picture processing and 

pattern recognition it is necessary to recognize and classify pertinent structures 
in a picture consisting of a multiplicity of single picture elements. An 
important step therefore consists in combining pertinent, contiguous picture 
elements into coherent picture segments. This technique is referred to as 
segmentation. Through their shape and texture properties the picture segments 
thereby foimd offer substantially more information for a subsequent 
classification than single picture elements, with this information being strongly 
dependent on the quality of segmentation. 



[0004] The essential criteria for the segmentation of picture elements are 

contiguity and so-called homogeneity criteria v^hich determine whether or not 
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a picture segment, following the combination of picture elements, is judged to 
be homogeneous. A homogeneity criterion determines whether or not 
combination is performed. In a homogeneity criterion for example specific 
texture properties or a homogeneity of features can be defined. 

[0005] One particular difficulty in segmentation of a picture is caused by 

the texture of picture regions which occurs in very many pictures in most 
variegated degrees. Characteristically, almost all object types are more or less 
textured. A like texture is, besides other properties for example seediness or 
grain, essentially characterized by a higher or lower degree of distribution of 
features, for example a gray value distribution or color distribution. Such 
distributions of feature values may be greater or smaller for various objects and 
may also overlap. 

[0006] For example in Fig. 17 a case is represented wherein altogether 

six picture objects each presenting a differently sized range of gray values 
ranging on a scale between 0 and 256 are shown, wherein the respective range 
of gray values of the picture objects represented above partly or entirely 
overlaps with the one of the picture objects represented below. 

[0007] This causes considerable difficulties in segmentation as on the 

one hand, textured picture objects, which in part present very different feature 
values, are to be segmented entirely, whereas on the other hand segmentation 
is performed as a general rule on the basis of similarities of features. Where the 
differences of features within an picture object are too large, segmentation will 
become problematic owing to similarity of features. At the same time it is 
generally difficult to separate two contiguous picture objects having 
overlapping feature distributions. 
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[0008] In many segmentation methods of the prior art the similarity or 

pertinence of picture elements and picture segments is determined for example 
by means of so-called one- or more-dimensional threshold techniques, wherein 
the feature space defined by the features of the picture elements is classified 
into partial ranges. The homogeneity criterion is satisfied in these known 
methods whenever the picture elements are contained in a same partial range. 
Thus for example the 256 gradations of gray in which a digitized picture is 
frequently present, may be classified into 16 regions each having 16 gray 
values. If two contiguous picture elements are in a same range of gray values, 
they are judged to be similar and are therefore segmented. This does, however, 
often solve the above described difficulties only very insufficiently. Even where 
certain objects are well described by the classification of the regions, this 
classification may fail altogether for other picture objects. 

[0009] Other methods attempt to combine picture segments based on 

specific predetermined texture features which leads to a texture-based 
segmentation. This works more or less well for specific textures in particular 
picture regions, however often worse in other picture regions. These methods 
at the same time necessitate beforehand knowledge about the picture and its 
textures, and furthermore do not enable any desired resolutions. 

[00010] Segmentation methods utilizing watershed transformation 

employ as the basis for segmentation a representation of the color gradients in 
the picture, begin segmentation in the most homogeneous picture regions, and 
successively, i.e. by and by, expand the segments into more heterogeneous 
picture regions. In this way homogeneous picture regions are well 
segmentable, uniformly heterogeneous picture regions however with more 
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difficulty. Simultaneous segmentation of homogeneous and heterogeneous 
picture regions, or of picture regions having different degrees of heterogeneity, 
can be performed only with great difficulty. In addition the method disregards 
the original color information. 

[00011] Techniques performing pixel classification operate along the 

principle of determining, based on beforehand knowledge, intervals or 
distributions in the feature space which are known to be characteristic for 
particular object classes. In this case the examined picture elements are each 
allocated to a respective class with which they present the highest probability 
of pertinence in accordance with their vector in the feature space. The 
homogeneity criterion is in this case defined as pertinence to the same object 
class. Apart from the fact that much less information for classification is 
available about single picture elements than about picture segments, a 
particular difficulty is encountered in processing aerial and satellite pictures 
owing to the fact that the distribution of the features for particular object types 
depends very strongly on weather, time of the day and season, as well as light 
conditions at the time when the picture was taken. Accordingly a suitable 
preliminary definition of typical distributions for particular object classes in the 
feature space is very difficult. 

[00012] From the article J.C. Tilton: "HYBRID SEGMENTATION FOR 

EARTH REMOTE SENSING DATA ANALYSIS", IEEE International, vol. 1, pp. 
703 to 705, there is known a method for segmentation of a digital picture which 
uses a combination of region growing and boundary detection. In a first step of 
this method edge boundaries are detected and in a second step of this method 
region growing is performed by not allowing to grow regions past edge 
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boundaries defined by the boundary detection. However, this method has 
disadvantages in that the maximum size of regions to be grown is the size 
defined by the edge boundaries. Thus, this method fails to provide the 
possibility to grow regions having similar features past edge boimdaries which 
are in fact no edges of different picture objects but result from textures or the 
like. 

SUMMARY OF THE INVENTION 

[00013] It is therefore one object of the present invention to provide a 

method for segmentation of a digital picture which ensures an excellent 
segmentation or object recognition even in cases where respective objects have 
overlapping feature ranges or are characterized by feature values which are 
liable to strongly vary under different conditions. 

[00014] In accordance with a first aspect of the present invention there is 

provided a method for segmentation of a digital picture consisting of a 

multiplicity of single picture elements comprising (a) determining if one of one 

and several features relating to contiguous picture objects comprising picture 

elements and picture segments are conforming or not conforming based on a 

specific homogeneity criterion by means of referencing a predetermined 

tolerance for each feature as a termination criterion, within which feature 

values relating to the contiguous picture objects in question may differ; (b) if 

one of one feature and several features relating to the contiguous picture 

objects are determined to be conforming then merging the conforming picture 

objects; and (c) repeating the resulting segmentation until the resulting 

segmentation converges in a stable or approximately stable condition in which 

no further contiguous picture objects are determined to be conforming. 

5 
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[00015] In a preferred embodiment of the present invention a feature 

difference to be compared in the homogeneity criterion is determined via 
heterogeneity introduced by merging two picture objects by determining a 
difference Ah^ between heterogeneities of respective picture objects weighted 

with the size of the respective picture objects after and before merging so that 
homogeneity is expressed by the formula 

Ah^ = (ni + n2)hnew - (^ihi + n2h2) < a 
wherein a is the predetermined tolerance, h^ and h2 are heterogeneities 
of the respective picture objects, nj and n2 are sizes of the respective picture 
objects and hj^^^ is the heterogeneity of a potentially newly formed picture 
object. 

[00016] In a further preferred embodiment of the present invention the 

heterogeneity of the potentially newly formed picture object is defined as 
standard deviation of color mean values of the respective picture objects as 
expressed by the formula 

Ah^ = Aa^ = (ni + n2)anew " (^^l^^l + i^2^2) < « 
wherein ai and 02 are standard deviations of the respective picture 
objects and cj^iew is the standard deviation of the potentially newly formed 
picture object. 

[00017] In a further preferred embodiment of the present invention a 

predetermined value for the standard deviation is used for small picture 
objects having a size of approximately one to five picture elements. 

[00018] In a further preferred embodiment of the present invention the 

heterogeneity of the potentially newly formed picture object is defined as the 
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variance of color mean values of the respective picture objects as expressed by 
the formula 

Ah^ = Avar^ = (nj + ^2)^^^new ' (nivarj + n2var2) < a 
wherein varj and var2 are variances of the respective picture objects and 
varj^e^ is the variance of the potentially newly formed picture object. 

[00019] In a further preferred embodiment of the present invention the 

heterogeneity of a potentially newly formed picture object is determined via a 
weighted difference of color mean values of the respective picture objects 
before and after merging as expressed by the formula 

Ah^ = Am^ = (ni + n2) I m^ew ' " (^^1 ' ^^1 1 + n2 I m2 I ) < a 
wherein mi and m2 are color mean values of the respective picture 
objects and mj^^^ is the color mean value of the potentially newly formed 
picture object. 

[00020] In a further preferred embodiment of the present invention 

picture regions having continuous color transitions are combined and the 
heterogeneity of the potentially newly formed picture object is defined as one 
of the average distance and the average value of the squares of the distances of 
the color mean values of the respective picture objects depending on the 
dimensionality of the topical space in relation to a regression line, surface or 
hypersurface of the color mean values in the topical space or in relation to 
another function of approximation to the color mean values in the topical 
space. 

[00021] In a further preferred embodiment of the present invention after 

step (c) the steps of (d) determining if a new tolerance is selected; and (e) if a 
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new tolerance is selected then repeating the method for segmentation and 
returning to step (a) to thereby form a hierarchical structure of picture objects 
having different hierarchy planes, are performed. 

[00022] In a further preferred embodiment of the present invention on a 

lowest hierarchy plane of the hierarchical structure picture elements are 
located, which are then at least on a next higher hierarchy plane of the 
hierarchical structure merged into over-picture objects, which may in turn be 
merged once or several times into over-picture objects on higher hierarchy 
planes of the hierarchical structure. 

[00023] In a further preferred embodiment of the present invention the 

highest hierarchy plane of the hierarchical structure contains only one picture 
object. 

[00024] In a further preferred embodiment of the present invention a new 

hierarchy plane is introduced in the hierarchical structure by initially 
duplicating all picture objects on a next lower hierarchy plane and inserting 
these picture objects in the new hierarchy plane as respective over-picture 
objects of the picture objects on the next lower hierarchy plane, wherein the 
picture objects on the new hierarchy plane which are determined to be 
conforming are merged in such a manner that only picture objects are merged 
which do not have different over-picture objects on a next higher hierarchy 
plane. 

[00025] In a further preferred embodiment of the present invention an 

order in which picture objects of a plane are processed is an order which 
ensures a maximum possible distance from already processed picture objects 
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and is a pseudo-stochastic order wherein in multiple, repeated runs on a 
hierarchical plane within one run each picture object present at the beginning 
of the run is processed once at the most by merging. 

[00026] In a further preferred embodiment of the present invention the 

method further comprises the steps of (f) determining if one of one and several 
features of already merged picture objects are still conforming or not still 
conforming based on the specific homogeneity criterion; and (g) if one of one 
feature and several features of the already merged picture objects are 
determined not to be still conforming then excluding the not still conforming 
picture objects. 

[00027] In a further preferred embodiment of the present invention steps 

(f ) and (g) are performed in addition to steps (a) and (b) in an arbitrary order or 
in parallel. 

[00028] In a further preferred embodiment of the present invention the 

method further comprises the steps of (h) determining if a boxmdary picture 
object of already merged picture objects located at a boundary of the already 
merged picture objects satisfies the homogeneity criterion with the already 
merged picture objects as well as with one of one and several contiguous 
picture objects; and (i) if the homogeneity criterion is satisfied with the already 
merged picture objects as well as with the one of one and several contiguous 
picture objects, allocating the boundary picture object to the picture object with 
which the homogeneity criterion is satisfied best. 

[00029] In a further preferred embodiment of the present invention steps 

(h) and (i) are performed in addition to at least one of steps (a) and (b) and 
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steps (f) and (g) in an arbitrary order or in parallel. 



[00030] 



In a further preferred embodiment of the present invention if the 



homogeneity criterion is satisfied with the already merged picture objects as 
well as with the one of one and several contiguous picture objects, feature 
distributions of the already merged picture objects and the one of one and 
several contiguous picture objects are calculated and based thereon, a 
pertinence of the boundary picture object is determined in such a manner that 
the boundary picture object is allocated to the one picture object wherein a 
feature value of the boundary picture object occurs most frequently or that the 
boundary picture object is allocated to a picture object in a probabilistic 
manner by calculating probabilities based on a frequency of occurrence of the 
feature value of the boundary picture object in the already merged picture 
objects and the one of one and several contiguous picture objects. 

[00031] In a further preferred embodiment of the present invention for 

the feature distributions respective histograms of the features of the already 
merged picture objects and the one of one and several contiguous picture 
objects are referred to. 

[00032] In a further preferred embodiment of the present invention if a 

boundary picture object is re-grouped from one picture object into another 
picture object, coherence of the one picture object is examined and in case of 
non-coherence of the one picture object the one picture object is divided into 
corresponding coherent picture objects formed by re-grouping. 

[00033] In a further preferred embodiment of the present invention the 

digital picture comprises a plurality of single channels having a different 
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information content and boundary picture objects are regrouped from one 
picture object into another picture object only if the pertinence to the other 
picture object averaged through all single channels is greater than the 
pertinence to the one picture object averaged through all single channels. 

[00034] In a further preferred embodiment of the present invention in 

merging or boundary correction object-related various homogeneity criteria are 
employed in accordance with specific features of the picture objects comprising 
compactness, size, boundary roughness, linearity, gradient of color 
development, and their classification. 

[00035] In a further preferred embodiment of the present invention 

merging is performed only if a feature difference defined in the homogeneity 
criterion for one of the picture objects is smallest in comparison with the 
contiguous picture objects and contained within the predetermined tolerance. 

[00036] In a further preferred embodiment of the present invention 

merging is performed only if for two picture objects a feature difference 
defined in the homogeneity criterion is smallest in comparison with the other 
contiguous picture objects and is contained within the predetermined 
tolerance. 

[00037] In a further preferred embodiment of the present invention 

merging is performed only if a feature difference defined in the homogeneity 
criterion is smallest in comparison with all other possible combinations of 
picture objects and is contained within the predetermined tolerance. 

[00038] In a further preferred embodiment of the present invention 

picture objects are processed in a pseudo-stochastic order. 
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[00039] In a further preferred embodiment of the present invention 

picture objects are processed in an order which ensures maximal possible 
distance from already processed picture objects. 

[00040] In a further preferred embodiment of the present invention 

several picture objects are processed simultaneously. 

[00041] In a further preferred embodiment of the present invention the 

digital picture comprises a plurality of single channels having a different 
information content and picture objects are combined only if the homogeneity 
criterion referred to is satisfied for each one of the channels. 

[00042] In a further preferred embodiment of the present invention the 

digital picture comprises a plurality of single channels having a different 
information content and wherein for the homogeneity criterion standard 
deviations, variances or mean values of color values of picture objects are 
added up or averaged through all channels, wherein the channels may be 
weighted differently. 

[00043] In a further preferred embodiment of the present invention 

boundary picture objects are not referred to for determination of properties of 
large picture objects. 

[00044] In a further preferred embodiment of the present invention the 

homogeneity criterion comprises texture features. 

[00045] In a further preferred embodiment of the present invention a 

segmentation of lines is performed in which a value of linearity indicating a 
ratio of length and width of a picture object is used as a shape feature and 
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picture objects are initially segmented such that each picture object exceeding a 
threshold for the value of linearity is processed as a line object. 

[00046] In a further preferred embodiment of the present invention 

processing of a picture object as a line object comprises the steps of searching 
for picture objects matching picture objects complementing a line in a direction 
of line ends beyond an immediate vicinity as far as a specific distance and in a 
sector having a specific angle; determining a factor which improves a matching 
value determined in the homogeneity criterion depending on how well a line is 
complemented by a previous line object, so that in case of an identical 
matching value linear combination is preferred to the usual combination; 
determining criteria for how well a line is complemented, such as an 
improvement of the linearity of the previous line object, in addition to at least 
one of identical color contrast with the surroundings and identical color; and 
establishing a connection having a minimum possible thickness between non- 
contiguous picture objects such that the line object is in any case diagonally 
coherent, if a matching picture object is not found in the immediate vicinity. 

[00047] In a further preferred embodiment of the present invention upon 

additionally performing linear segmentation, segmentation is initially 
performed with diagonal vicinity, wherein small and linear picture objects 
have diagonal coherence and all other picture objects have planar coherence, 
wherein, if a hitherto small, diagonally picture object exceeds a critical size or if 
a hitherto linear picture object drops below a critical linearity by area merging, 
it is divided into its components having planar coherence. 

[00048] In accordance with a second aspect of the present invention there 

is provided a method for segmentation of a digital picture consisting of a 
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multiplicity of single picture elements comprising (a) determining if one of one 



contiguous picture objects in question as a continuation criterion which leads 
to a minimum increase in a defined value for the complexity of an entire 
structure consisting of all picture objects; (b) if one of one feature and several 
features relating to the contiguous picture objects are determined to be 
conforming then combining the conforming picture objects; and (c) repeating 
the resulting segmentation until the resulting segmentation converges in a 
stable or approximately stable condition in which no further contiguous 
picture objects are determined to be conforming. 

[00049] In a further preferred embodiment of the present invention by 

means of repeating the resulting segmentation a hierarchical structure is 
formed having several hierarchical planes which are present in a locally 
different hierarchical depth. 

[00050] In a further preferred embodiment of the present invention a 

highest hierarchical plane consists of a single picture object containing all 
picture elements. 

[00051] In a further preferred embodiment of the present invention 

respective modifications are performed on respective highest local hierarchical 
planes of the hierarchical structure. 

[00052] In a further preferred embodiment of the present invention the 

modifications comprise at least one of merging two contiguous picture objects. 



and several features relating to contiguous picture objects comprising picture 
elements and picture segments are conforming or not conforming based on a 
specific homogeneity criterion by means of determining a modification of the 
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exclusion of a picture object from another picture object, allocating a boundary 
picture object located at the boundary of already merged picture objects to 
another contiguous picture object and founding a new picture object on a next 
higher local hierarchical plane to be formed. 

[00053] In a further preferred embodiment of the present invention the 

defined value for the complexity is defined as one of the sum through all 
picture objects in the hierarchical structure of standard deviations of color 
mean values of objects multiplied by the number of picture objects on a next 
lower hierarchical plane in each picture object 

C = ^ 

i Udir Udir 

[00054] wherein C is the defined value for the complexity, a iu^j^. is the 

standard deviation of the color mean value of a respective picture object i, and 
^iUdir number of the picture objects on the next lower hierarchical plane 

in each picture object, and the sum through all picture objects in the 
hierarchical structure of variances of color mean values of the picture objects 
on the next lower hierarchical plane in each picture object multiplied by the 
number of picture objects on the next lower hierarchical plane in each picture 
object 

C = n. 

van... • 
1 Udir Udir 

wherein C is the defined value for the complexity, vari^^^j, is the 

variance of the color mean value of a respective picture object i, and rii^^^j. is 

the number of the picture objects on the next lower hierarchical plane in each 
picture object. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

[00055] These and other aspects of the present invention will be apparent 

from and elucidated with reference to the embodiments described hereinafter 
and considered in connection with the accompanying drawings, in which: 

[00056] FIG. 1 illustrates a flow chart of a method for segmentation of a 

digital picture in accordance with an embodiment of the present invention; 

[00057] FIG. 2 illustrates an operation "merging" performed in the 

method in accordance with the embodiment of the present invention; 

[00058] FIG. 3 illustrates a flow chart of an exclusion routine which can be 

performed in the method in accordance with the embodiment of the present 
invention; 

[00059] FIG. 4 illustrates an operation "exclusion" performed in the 

routine shown in FIG. 3; 

[00060] FIG. 5 illustrates a flow chart of a boundary correction routine 

which can be performed in the method in accordance with the embodiment of 
the present invention; 

[00061] FIG. 6 illustrates an operation "boimdary correction" performed 

in the routine shown in FIG. 5; 

[00062] FIG. 7 illustrates a hierarchical structure formed by the method in 

accordance with the embodiment of the present invention; 

[00063] FIG. 8 illustrates a flow chart of a method in accordance with a 

modification of the embodiment of the present invention; 
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[00064] FIG. 9 illustrates a flow chart of a merging routine which can be 

performed in the method in accordance with the modification of the 
embodiment of the present invention; 

[00065] FIG. 10 illustrates an operation "merging" performed in the 

routine shown in FIG. 9; 

[00066] FIG. 11 illustrates a flow chart of an exclusion routine which can 

be performed in the method in accordance with the modification of the 
embodiment of the present invention; 

[00067] FIG. 12 illustrates an operation "exclusion" performed in the 

routine shown in FIG. 11; 

[00068] FIG. 13 illustrates a flow chart of a boundary correction routine 

which can be performed in the method in accordance with the modification of 
the embodiment of the present invention; 

[00069] FIG. 14 illustrates an operation "boimdary correction" performed 

in the routine shown in FIG. 13; 

[00070] FIG. 15 illustrates an operation "founding" which can be 

performed in the method in accordance with the modification of the 
embodiment of the present invention; 

[00071] FIG. 16 illustrates a hierarchical structure formed by the method 

in accordance with the modification of the embodiment of the present 
invention; and 
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[00072] 



FIG. 17 illustrates gray values of six picture objects each 



presenting a differently sized range of gray values v^ithin a scale of gray values 



between 0 and 256. 



DETAILED DESCRIPTION OF THE INVENTION 



[00073] 



First of all, it should be noted that the term "digital picture" as 



used herein should be understood to signify a digital data record having any 
dimensionality to be segmented and the term "picture element" should be 
imderstood to signify a smallest single element contained in this data record, 
such as for example a pixel or a voxel. Furthermore, it should be noted that the 
term "picture object" as used herein should be understood to jointly signify a 
picture element or a picture segment and that a picture segment may equally 
contain a single picture element. 

[00074] It should be noted that the present invention is applicable on a 

digital picture consisting of a multiplicity of single picture elements. The 
method of the present invention can be applied on a digital picture v^hich has 
not been processed before and thus contains only picture elements as picture 
objects as well as on a digital picture which has been processed before and thus 
contains picture elements as well as picture segments as picture objects, to 
thereby again process the digital picture under different conditions or the like. 
The fundamental functional principle of the present invention is to process 
contiguous picture objects, i.e. picture elements or picture segments, which are 
determined to be conforming based on a homogeneity criterion for one or 
several features relating to contiguous picture objects. Processing of picture 
objects includes merging of picture objects and some other possibilities of 
processing picture objects as will be described below in more detail. 
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[00075] Referring now to FIG. 1, there is illustrated a flow chart of a 

method for segmentation of a digital picture in accordance with an 
embodiment of the present invention. 

[00076] In the first step SI 00 a homogeneity criterion is selected. Such 

homogeneity criterion defines the condition under which it is determined 
whether one or several features relating to contiguous picture objects are 
conforming or not conforming. In the next step SI 10 a kind of matching is 
selected. This means that it is selected whether the homogeneity criterion 
should be applied to contiguous picture objects by means of using simple 
matching, best matching, mutual best matching or global best matching. In the 
next step S120 a tolerance for the homogeneity criterion is selected. In more 
detail, the tolerance of the homogeneity criterion is referenced for each feature 
of contiguous picture objects as a termination criterion within which feature 
values relating to contiguous picture objects may differ: 

[00077] Details of the homogeneity criterion, the Icind of matching and 

the tolerance for the homogeneity criterion will now be described. There are 
various possibilities which do, however, all take into consideration the specific 
distribution of features or other properties such as the size of picture objects, 
texture, shape, orientation or the like. 

[00078] In general, two different types of homogeneity criteria are used to 

determine a difference between feature values of contiguous picture objects. In 

the first type of homogeneity criteria, the difference in feature values between 

two contiguous picture objects is compared directly. In the second type of 

homogeneity criteria, a potential condition of two contiguous picture objects 

after merging is compared with the condition before merging to thereby 
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determine the heterogeneity introduced by a facultative combination. Thus, it 
is possible to deduct the difference between feature values of the two 
contiguous picture objects. 

[00079] In the following, the first type of homogeneity criteria will be 

described in further detail. 

[00080] In the most simple case, the color mean value in accordance with 

the following inequation (1) enters in the homogeneity criterion which 
describes the conformity or similarity of two contiguous picture objects 1 and 
2. 

I mi - m2 I < a (1) 

[00081] In inequation (1) a designates a predetermined factor globally 

determining the tolerance for each feature relating to contiguous picture objects 
for the entire digital picture and mi and m2 designate color mean values of the 

two contiguous picture objects 1 and 2. Inequation (1) specifies that the 
difference of the color mean values m^ and m2 must be smaller than a globally 
predetermined tolerance, i.e. the value a. 

[00082] The accuracy of segmentation can be further improved if the 

homogeneity criterion of inequation (1) is modified in such a way that in 
addition the standard deviation of feature distributions of the two contiguous 
picture objects is taken into consideration, as can be seen from the following 
inequation (2). 



<a (2) 
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[00083] 



In inequation (2) a again designates the tolerance, ai and 02 



designate standard deviations of the two contiguous picture objects 1 and 2 



picture objects 1 and 2. Inequation (1) specifies that the difference of the color 
mean values and m2, divided by the sum of the standard deviations ai and 
a2, must be smaller than the globally predetermined tolerance, i.e. the value a. 
The standard deviations and 02 establish a local object relations which leads 

thereto that, the more heterogeneous the two contiguous picture objects 1 and 2 
already are, the more tolerantly the difference in the color mean values mi and 
m2 is evaluated. 

[00084] As small picture objects of approximately 1 to 5 picture elements 

do not have, or do not yet have, a representative standard deviation, a specific 
standard deviation can instead be used for such small picture objects. The 
specific standard deviation for such small picture objects can be set to a 
predetermined value or can also be derived for example from the standard 
deviation of the entire digital picture or from a corresponding context. 

[00085] Furthermore, the merging of two contiguous picture objects 

depends on current properties of the two contiguous picture objects and thus 
on the degree in which the two contiguous picture objects have already been 
segmented. Namely, the homogeneity criterion in accordance with inequation 
(1) or (2) also signifies that the similarity of a picture object with another one 
will always change with its current composition. In other words, the 
segmentation in a method which repeatedly redefines the properties in relation 
to an object is a historical method. Starting with a picture object by successively 



and mi and m2 again designate color mean values of the two contiguous 
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merging with other picture objects changes the properties of the thus formed 
new picture object and thus the conditions for merging with a next following 
picture object. Hence, whether or not a particular picture object is merged with 
another particular picture object depends on what properties the particular 
picture object previously has, and this in turn depends on the size of the 
particular picture object and the properties of the picture elements contained in 
the particular picture object. 

[00086] In the following, example are given to indicate what additional 

influence the size of the picture object exerts in the merging of picture objects. 

[00087] Starting out from a relatively large picture object to be 

segmented, small partial picture object at the outset of the segmentation 
method correspond to small samples and larger partial picture objects 
correspond to larger samples from the overall distribution of the relatively 
large picture object. 

[00088] The smaller the partial picture objects are, the larger is the 

average deviation of its mean values from the average distribution value of the 

picture object. In partial picture objects consisting of merely a single picture 

element, this is by definition the standard deviation of the overall distribution 

of the picture object. The distribution of larger partial picture objects, in turn, 

already corresponds very much better to the overall distribution of the picture 

object. In general, this means that, based on distributions of feature values of 

larger partial picture objects, much more safe and accurate decisions can be 

made than based on distribution of feature values of smaller partial picture 

objects. At the same time, the error which results from wrongly merging large 

picture objects is substantially larger than the error which results from wrongly 

22 



n . e 




min 








^1 


+ ^2 



003400.P009X Express M^^o.: EL634499961US 

merging small picture objects. 

[00089] Binding tolerances between large picture objects should 

therefore, at otherwise identical distribution properties, be more strict than 
between small picture objects. Consequently, the size of the picture objects 
must be taken into consideration in merging. This is achieved for example by a 
further differentiation of inequation (2) as can been seen from the inequation 
(3) below. 



< a (3) 



[00090] In inequation (3) nj^^j^ designates the size of the smaller one of 

two contiguous picture objects by means of the number of the picture elements 
and e designates a globally predetermined parameter which indicates how 
strongly this size is to influence the homogeneity criterion. If £ = 0, the size of 
the two contiguous picture objects is not taken into consideration. The larger e 
is selected, the larger the difference between the binding tolerance in merging 
smaller picture objects and the strictness in merging larger picture objects 
becomes. 

[00091] The following is an example of a criterion which evaluates the 

difference in the mean values and in the standard deviations. In a similar 
manner as in inequation (2), the similarity of the standard deviations can 
additionally be included in the homogeneity criterion. The homogeneity 
criterion is then obtained as the following relation (4) containing two 
inequations which must be satisfied at the same time (a is the logical AND 
operator). 
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<P (4) 



[00092] In relation (4) p designates a globally predetermined parameter, 

i.e. a further tolerance, and nj and 1x2 designate the number of the picture 
elements of the two contiguous picture objects. 

[00093] In the following, the second type of homogeneity criteria will be 

described in further detail. 

[00094] By using this second type of homogeneity criteria even better 

results may be obtained. Homogeneity criteria of the second type are those 
which determine the difference between feature values of two contiguous 
picture objects by the heterogeneity introduced into the segmentation through 
merging of two contiguous picture objects by comparison of the conditions 
after and before merging. The smaller the introduced heterogeneity is, the 
smaller is also the difference in feature values of the two contiguous picture 
objects. The size, i.e. the number of the picture elements of the respective 
picture objects is also taken into consideration by weighting. Instead of 
attempting to merge two contiguous picture objects having maximum possible 
homogeneity by direct comparison of features between the two contiguous 
picture objects, the actual condition after merging is compared with the one 
before merging by means of the second type of homogeneity criteria. Thus, it is 
ensured that as little as heterogeneity as possible is introduced into the 
segmentation. 

[00095] In general, a homogeneity criterion of the second type can be 

determined as the heterogeneity Ah^ of a potentially newly formed picture 
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object via a weighted difference of color mean values of respective picture 
objects before and after merging as shown in inequation (5) below. 

Ah^ = (ni + n2)hnew " (^ihi + n2h2) < a (5) 

[00096] In inequation (5) a is the predetermined tolerance, hj and h2 are 

heterogeneities of the respective picture objects, n^ and n2 are sizes of the 

respective picture objects and hj^^^ is the heterogeneity of the potentially 

newly formed picture object. Thus, Ah^ is the difference between the 

heterogeneities of the respective picture objects weighted with the size of the 
respective picture after merging and before. 

[00097] In more detail, this second type of homogeneity criterion as 

shown in inequation (5) can be defined as follows. 

[00098] The heterogeneity of the potentially newly formed picture object 

can defined as weighted standard deviation of color mean values of the 
respective picture objects as shown in inequation (6) below. 

Ahw = Aa^ = (ni + n2)anew " (^1^1 + ^2^2) < a (6) 

[00099] In inequation (6) ai and 02 are standard deviations of the 

respective picture objects and a^ew standard deviation of the potentially 

newly formed picture object. In this case, a predetermined value for the 
standard deviation can be used for small picture objects having a size of 
approximately one to five picture elements. 

[000100] The heterogeneity of the potentially newly formed picture object 

can be defined as the weighted variance of color mean values of the respective 
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picture objects as shown in inequation (7) below. 

Ah^ = Avar^ = (ni + n2)varnew " (7) 
(nj^var]^ + n2var2) < a 

[000101] In inequation (7) var^ and var2 are variances of the respective 
picture objects and var^^^^ is the variance of the potentially newly formed 
picture object. 

[000102] The heterogeneity of a potentially newly formed picture object 
can be determined via a weighted difference of color mean values of the 
respective picture objects before and after merging as shown in inequation (8) 
below. 

Ah^ = Am^ = (ni + n2) I m^ew I " (8) 
(nj I mi I + n2 1 m2 1 ) < a 

[000103] In inequation (8) m^ and m2 are color mean values of the 

respective picture objects and mj^^^ is the color mean value of the potentially 
newly formed picture object. 

[000104] In more detail, as shown in inequations (6) to (8) the second type 
of homogeneity criterion includes the comparison of the weighted standard 
deviation, the comparison of the weighted mean value modification and the 
comparison of the weighted variance after and before merging, respectively. 

[000105] Another homogeneity criterion of the second type is suited for 
segmenting picture regions having continuous color transitions. The 
heterogeneity of the potentially newly formed picture object is defined as one 
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of the average distance and the average value of the squares of the distances of 
the color mean values of the respective picture objects depending on the 
dimensionality of the topical space in relation to a regression line, surface or 
hypersurface of the color mean values in the topical space or in relation to 
another function of approximation to the color mean values in the topical 
space. To this end, for example in a two-dimensional picture the average 
distance or the averaged square distance of the color mean values of partial 
picture objects, such as for example the picture elements, from a regression 
surface of these color mean values in the topical space is determined using 
complex functions. This is then the measure for the heterogeneity of the picture 
object and can be used in inequation (5). In this manner, picture objects having 
a maximum continuous change or no change of color over the regression line 
are formed. Depending on necessity, the color distribution on the topical space 
can also be approximated by linear functions instead of by complex functions. 
Such homogeneity criterion is, for example, suited for segmenting two- 
dimensional photographic representations of three-dimensional objects and the 
color developments created by such representation into suitable picture objects. 

1000106] The kind of matching will now be described in further detail. 

[000107] In general, simple matching, best matching, mutual best 
matching and global best matching are used. 

[000108] By using simple matching merging is performed if a feature 

difference defined in the homogeneity criterion is contained within the 

predetermined tolerance. By using best matching merging is performed only if 

a feature difference defined in the homogeneity criterion for one of contiguous 

picture objects is smallest in comparison with the other contiguous picture 
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objects and contained within the predetermined tolerance. By using mutual 
best matching merging is performed only if for two contiguous picture objects 
a feature difference defined in the homogeneity criterion is smallest in 
comparison with all other contiguous picture objects and is contained within 
the predetermined tolerance. Finally, by using global best matching merging is 
performed only if a feature difference defined in the homogeneity criterion is 
smallest in comparison with all other possible combinations of picture objects 
and is contained within the predetermined tolerance. 

[000109] In the order of simple matching, best matching, mutual best 
matching and global best matching the quality of the segmentation 
successively increases as has been found by the inventors of the present 
invention by using the average weighted standard deviation of the average 
color mean values of the picture objects or the average weighted variance of 
the color mean values of the picture objects as a measure for the homogeneity 
and thus for the quality of the segmentation. 

[000110] By employing mutual or global best match, a development of 

segmentation is ensured which initially starts in picture regions having a 
maximum possible degree of homogeneity, continues via ever more 
heterogeneous picture objects, until further merging of picture objects is 
prevented due the threshold defined by the predetermined tolerance. In more 
detail, similarly to watershed transformation methods, segmentation is started 
in picture regions having the highest degree of homogeneity, however picture 
regions having a higher degree of heterogeneity are subsequently also 
segmented and set apart from each other and from homogeneous ones. 



[000111] As to the predetermined tolerance it should be noted that 
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pertinence of picture objects is defined in such a way that the feature difference 
of two contiguous picture objects to be merged, which is determined in the 
homogeneity criterion, must be contained within the predetermined tolerance. 

[000112] Referring again to FIG. 1, in the next step S130 a picture object is 
selected. Although only one picture object is selected in step S130 it is also 
possible to select several picture objects at the same time. Thus, it is possible to 
generate segmentation kernels in a maximum possible number of locations of 
the picture region to be segmented. Such segmentation kernels. The higher the 
number of locations at which the segmentation is started at the same time, i.e. 
the higher the number of kernels available for the segmentation, the higher is 
the differentiation possible in performing the segmentation. For this reason it is 
advantageous to use an order in processing the picture objects which ensures a 
maximum possible geographical distance from already processed picture 
objects. Such order can be pseudo-stochastic, i.e. random. It should be noted 
that the order of processing of the picture objects is fundamentally 
independent of the type of the homogeneity criterion. By means of these 
measures it is achieved that the segmentation leads to very good results even 
where a picture is characterized by being highly textured and contiguous 
picture objects present overlapping feature ranges. 

[000113] In the next step S140 it is determined in the whether the selected 
picture object and one of the picture objects contiguous thereto are conforming 
or not conforming based on determining if a difference in feature values 
between the selected picture object and the picture objects contiguous thereto is 
contained within the selected tolerance of the selected homogeneity criterion 
and if the selected kind of matching is satisfied. By means of this at most one of 
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the contiguous picture objects and the selected picture object are determined to 
be conforming. If it is determined in step S140 that the selected picture object 
and one of the contiguous picture objects are conforming, step S150 is carried 
out to merge these picture objects in order to form a new picture object and, 
then step S160 is carried out. If it is determined in step S140 that the selected 
picture object does not conform with any of the contiguous picture objects, step 
S160 is directly carried out. 

[000114] FIG. 2 illustrates an operation "merging" which is performed in 
step S150 in FIG. 1. As is obvious from FIG. 2 a picture object '1" having a 
picture object "a" as a sub-picture object and a picture object "2" having picture 
objects "b" and "c" as sub-picture objects are merged to form a new picture 
object "1+2" having picture objects "a", "b" and "c" as sub-picture objects. 

[000115] In step S160 it is determined whether more picture objects should 
be processed. It should noted that each picture object which is present at the 
outset of a run of steps S130 to S160 is selected once at the most and preferably 
once. If it is determined in step S160 that more picture objects must be 
processed, the routine returns to step S130. If it is determined in step S160 that 
all picture objects have been processed, step SI 70 is carried out to determine 
whether there is a stable condition in which no further contiguous picture 
objects are determined to be conforming, i.e. in which no further merging of 
picture objects occurs and the segmentation result converges and remains 
unchanged despite further processing. If it is determined in step SI 70 that 
there is no stable condition, the routine returns to step SI 30. If it is determined 
in step S170 that there is a stable condition, step S180 is carried out to 
determine whether a further processing on another plane of a hierarchical 
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structure of picture objects is to be performed. If it is determined that no 
further processing is to be performed, the method is completed. If it is 
determined that a further processing on another plane is to be performed, the 
routine returns to step SIOO. 

[000116] The formation of such hierarchical structure will now be 
described in further detail. 

[000117] It is now assumed that there is a digital picture containing picture 
elements and that no further picture objects are formed. 

[000118] After performing steps SIOO to S120 to thereby select the 
homogeneity criterion, the kind of matching and the tolerance for the 
homogeneity criterion, each picture element is duplicated on a new 
hierarchical plane. Then, each picture object is allocated a corresponding 
picture element as an over-picture object. Thus, the new hierarchical plane has 
already formed entirely. Carrying out steps S130 to S160 in multiple, repeated 
runs until there is a stable condition as mentioned above results in merging of 
picture objects on the new hierarchical plane to thereby form picture objects 
having one or several picture elements as sub-picture objects. Thus, there is 
provided a hierarchical structure in which the picture elements are present as a 
lowest hierarchical plane. 

[000119] Thus, by means of multiple repetitions of steps S130 to SI 60 one 
hierarchical plane of a hierarchical structure is formed. If it is determined that a 
further processing on another plane is to be performed, steps SIOO to S120 are 
carried out again and in step SI 20 a new tolerance for the homogeneity 
criterion is selected. Such selection of a new tolerance sets a new resolution for 
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the segmentation and thus results in the formation of a new hierarchical plane. 



hierarchical structure is formed and if a smaller tolerance is selected a lower 
hierarchical plane is formed. If a tolerance is selected which is larger than a 
tolerance of a hierarchical plane already formed and lower than a tolerance of 
another hierarchical plane already formed, a new hierarchical plane is inserted 
between these already formed hierarchical planes. 

[000121] By means of the selected tolerance for the homogeneity criterion 
it is determined on which hierarchical level a new hierarchical plane is to be 
inserted into an already existing hierarchical structure of picture objects. Then, 
each picture object on a next lower hierarchical plane with respect to the new 
hierarchical plane is removed from its previous over-object and is duplicated 
on the new hierarchical plane. Then the picture object on the next lower 
hierarchical plane is allocated the duplicated picture object as an over-object 
and the duplicated picture object is inserted into the previous over-picture 
object of the picture object on the next lower hierarchical plane as a sub-picture 
object. Thus, the new hierarchical plane has already formed entirely. Carrying 
out steps S130 to S160 in multiple, repeated runs imtil there is a stable 
condition as mentioned above results in merging of picture objects on the new 
hierarchical plane to thereby form picture objects having sub-picture objects. 
Thus, there is provided a hierarchical structure in which the picture elements 
are present as a lowest hierarchical plane. It should be noted that only such 
contiguous picture objects on the new hierarchical plane can be merged which 



The resolution of the segmentation can be expressed as the average number of 



picture elements per segment. 



[000120] 



If a larger tolerance is selected, a higher hierarchical plane of the 
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have the same over-picture object. 



[000122] 



By repeating the method of segmentation by means of using 



different tolerances a hierarchical structure is formed as shown in FIG. 7. 



hierarchical planes 1 to 4. on the lowest hierarchical plane, which is the first 
hierarchical plane 1, picture elements 5 are located, which are merged on the 
second hierarchical plane 2 to form picture objects 6 having the picture 
elements 5 as sub-picture objects. Furthermore, the picture objects 6 on the 
second hierarchical plane 2 are merged on the third hierarchical plane to form 
picture objects 6 having the picture objects 6 on the second hierarchical plane 
as sub-picture objects and so on. 

[000124] The highest hierarchical plane of the hierarchical structure can 
obtain only one picture object, if the tolerance is set to an appropriate large 
value. 

[000125] By means of the hierarchical structure picture information is 
concurrently present in a segmented condition in various degrees of resolution. 

[000126] Once a hierarchical structure is formed, the order of processing 
can be an arbitrary order. For example, the order of processing can start on the 
lowest possible hierarchical plane torn then stepwise process higher 
hierarchical planes or the order of processing may start on the highest possible 
hierarchical plane to then process stepwise lower hierarchical planes or the 
roder of processing may be any desired one and a new hierarchical plane are 
also inserted between two other hierarchical planes. 



[000123] 



FIG. 7 illustrates a hierarchical structure having first to fourth 
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[000127] There has been described the operation of merging. However 
other operations can be used to process the picture objects and to improve the 
result of the segmentation. Such operations comprise the operations 
"exclusion" and "boundary correction". 

[000128] FIG. 3 illustrates a flow chart of an exclusion routine which can be 
performed in the method in accordance with the embodiment of the present 
invention. 

[000129] It should be noted that this exclusion routine can be performed in 
addition to steps S130 to S160 of FIG. 1 or in parallel in an arbitrary order. 

[000130] As shown in FIG. 3 in step S200 a picture object is selected. The 
selection is performed in the same manner as described with reference to step 
S130 in FIG. 1 and thus a detailed description is omitted here. In step S210 it is 
determined whether picture objects which have been merged to form the 
selected picture object still conform. The determination of step S210 is the same 
as the determination of step S140 of FIG. 1 and thus a detailed description is 
omitted here. If it is determined in step S210 that the merged picture objects 
still conforming step S230 is carried out. If it is determined that the merged 
picture objects do not still conforming then step S220 is carried out. In step 
S220 non-conforming partial picture objects of the selected picture object are 
excluded from the selected picture object. In step S220 it is determined whether 
more picture objects are to be processed. The determination of this step is the 
same as the determination of step S160 in FIG. 1 and thus a detailed description 
is omitted here. After step S230 the exclusion routine is completed. 

[000131] FIG. 4 illustrates an operation "exclusion" which is performed in 
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step S220 in FIG. 3. As is obvious from FIG. 4 picture object "1+2" having 
picture objects "a", "b" and "c" as sub-picture objects is divided by exclusion to 
form new a picture object "1" having picture object "a" as sub-picture object and 
a new picture object "2" having picture objects "b" and "c" as sub-picture 
objects. 

[000132] FIG. 5 illustrates a flow chart of a boundary correction routine 
which can be performed in the method in accordance with the embodiment of 
the present invention. 

[000133] It should be noted that this boundary correction routine can be 
performed in addition to at least one of steps S130 to S160 of FIG. 1 and steps 
S200 to S230 of FIG. 3 or in parallel in an arbitrary order. 

[000134] As shown in FIG. 5 in step S300 a picture object is selected. The 
selection is performed in the same manner as described with reference to step 
S130 in FIG. 1 and thus a detailed description is omitted here. In step S310 it is 
determined whether for boundary picture objects of the selected picture object 
which are sub-picture objects of the selected picture object on the next lower 
hierarchy plane the homogeneity criterion does match better with other 
contiguous picture objects of the selected picture object. If it is determined in 
step S3 10 that the homogeneity does not match better with other contiguous 
picture objects of the selected picture object step S330 is carried out. If it is 
determined that the homogeneity criterion does match better with one of the 
contiguous picture objects of the selected picture object step S320 is carried out 
to exchange the boundary picture objects to now be a new boundary picture 
object of the contiguous picture object of the selected picture object with which 
the homogeneity criterion is satisfied best. 
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[0001351 



The determination and exchange carried out in steps S310 and 



S320 can be carried out such that if the homogeneity criterion is satisfied with 
the already merged picture objects as well as with one of one and several 
contiguous picture objects, feature distributions of already merged picture 
objects and the one of one and several contiguous picture objects are calculated 
and based thereon, a pertinence of the boundary picture object is determined in 
such a manner that the boimdary picture object is allocated to the one picture 
object wherein a feature value of the boundary picture object occurs most 
frequently or that the boundary picture object is allocated to a picture object in 
a probabilistic maimer by calculating probabilities based on a frequency of 
occurrence of the feature value of the boundary picture object in the already 
merged picture objects and the one of one and several contiguous picture 
objects. For example, for the feature distributions respective histograms of the 
features of the already merged picture objects and the one of one and several 
contiguous picture objects are referred to. 

[000136] Then step 320 is carried out. In step S320 it is determined whether 
more picture objects are to be processed. The determination of this step is the 
same as the determination of step S160 in FIG. 1 and thus a detailed description 
is omitted here. After step S330 the boimdary correction routine is completed. 

[000137] FIG. 6 illustrates an operation "boundary correction" performed 
in the routine shown in FIG. 5. 

[000138] As a criterion for 



[000139] As shown in FIG. 6 a boundary picture object "d" of picture object 
"1" is exchanged between picture object "1" and picture object "2" to become a 
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new boundary picture object of picture object "2" and a boundary picture object 
"i" of picture object "3" is exchanged between picture object "3" and picture 
object "4" to become a new boundary picture object of picture object "4". 

[000140] Segmentation with homogeneity criteria that are adapted in an 
object-related manner can be performed in accordance with features or 
classification. Various homogeneity criteria can be employed in accordance 
with specific features of the picture objects comprising compactness, size, 
boundary roughness, linearity, gradient of color development and their 
classification. 

[000141] In the exclusion and boundary correction routines described 
above coherence of picture objects must be ensured. When a picture object is 
sorted out in a thin location of its over-picture object due to exclusion or 
boimdary correction, there is the possibility that the over-picture object loses its 
direct vicinity coherence. In this case, the over-picture object must be divided 
into corresponding coherent picture objects. 

[000142] The fact that a picture object is modified by merging, exclusion 
and boundary correction results in a modification of its properties and thus of 
the homogeneity criterion. Picture objects previously not having matched this 
picture object may now match this picture object. Vice versa picture objects 
which once were incorporated into this picture object may now not satisfy the 
now modified homogeneity criterion with this picture object any more. In this 
way, continuous picture objects are merged, botmdaries of picture objects are 
shifted or picture objects not matching any more are excluded from a over- 
picture object. 
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[000143] 



The dependency of the respective segmentation status on the 



specific development of the picture or of the picture object, i.e. the historicity of 
the method, is herein consciously accepted. The iterative, cyclical optimization 
of the picture has the result that the final state is reproducibly of the same kind 
although small differences may occur in details. This does, however, not affect 
the high quality of the final results. 

[000144] There is also the possibility that boundary picture objects are not 
referred to for the determination of properties of large picture objects. 

[000145] In the following, examples for processing the homogeneity 
criterion in segmentation through several channels shall be explained. The 
term "channel" should in this context be vinderstood to indicate that the 
respective picture consists of several single pictures, i.e. channels, having 
different information contents. 

[000146] Where a digital picture is available in a plurality of channels, the 
question is how an overall decision can be made based on the examination of 
the homogeneity criterion, which occur in every decision about each channel. 

[000147] On the one hand, it is possible to refer to an absolute criterion. 
Unless the homogeneity criterion is satisfied in each channel, combining of 
picture objects is not performed. Hereby good utilization is ensured. Even if 
the homogeneity criterion is satisfied in many channels, the difference in 
another channel nevertheless signifies that the underlying picture structures 
are different. 

[000148] On the other hand, it is also possible to sum up or average the 
differences of features such as standard deviations, variances or mean values of 
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color values of picture objects which the homogeneity criterion determines in 
each channel as a quality criterion through all channels. The channels may for 
this purpose be weighted differently. If a quality criterion is employed in the 
overall segmentation which also averages through the charmels, then summing 
up or averaging in application of the homogeneity criterion leads to better 
results than an absolute criterion. 

[000149] In bovindary correction the overall probability of pertinence 
results in the geometrical average of the single probability of pertinences in 
every channel. This means that up to a certain degree, a worse probability of 
pertinence in a channel can be compensated by a good one in another one. 
Nevertheless when the pertinence in a charmel is particularly bad, this is 
expressed clearly in geometrical averaging despite good pertinence in other 
channels. 

[000150] The following description of the method refers to the 

segmentation of lines which can be additionally performed. 

[000151] At the outset, in accordance with the above description, 

segmentation is performed, with one of the homogeneity criteria of the second 

type which examines the feature difference between two picture objects via the 

heterogeneity introduced in a potential combination being used in mutual best 

match. For each processed picture object the best matching contiguous picture 

object is searched and these are combined only if the best match is actually 

mutual. At the same time, a "linearity" feature is defined which, for example, 

indicates the ratio of length and width of the picture object. Each processed 

picture segment is examined as to its linearity. When the linearity exceeds a 

specific predetermined value, the segment is processed, in addition to the 

39 



003400.P009X Express Mail N(^^L634499961US 



above described manner of proceeding, as a line object. I.e. in addition to the 
directly adjacent neighbours, matching picture objects complementing the line 
are also searched in the direction of the line ends as far as a predetermined 
distance and in a sector of a predetermined angle. In the degree in which the 
line is complemented in a combination, a factor f (0 < f < 1) is reduced, with the 
feature difference in the homogeneity criterion being additionally reduced with 
this factor. In this manner matching with picture objects complementing the 
line is comparable to matching with normal picture objects. A linear picture 
object can, depending on mutual best match, be combined with contiguous 
picture objects in such a way that linearity is increased, however also be 
combined in the usual manner, with linearity decreasing as a consequence. 

[000152] Where a picture object not located in directly contiguous vicinity 
and complementing a line is foimd in best match, it is possible to draw a 
connecting line having a minimum thickness which ensures coherence. The 
eligible picture objects must then possibly be sorted out from already existing 
picture objects and subsequently be allocated to the line object. 

[000153] In the case of very thin line structures it can additionally be of 
advantage to admit a diagonal vicinity for small and for linear picture objects, 
whereas for all other picture objects only planar but not diagonal coherence is 
possible. Hereby lines and line Icemels can be created more easily in the 
segmentation. When a hitherto small, diagonally coherent picture object 
exceeds a critical size or when a hitherto linear picture object drops below a 
critical linearity as a result of merging, it is divided into its components having 
planar coherence. 

[000154] Referring now to FIG. 8, there is illustrated a flow chart of a 
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method for segmentation of a digital picture in accordance with a modification 
of the embodiment of the present invention. 



invention will be described by means of describing merely the modifications 
with respect to the embodiment of the present invention. All other features 
described above with respect to the embodiment of the present invention are 
also applicable on the modification of the embodiment of the present 
invention. 

[000156] In steps S400 and S410 in the same manner as described with 
respect to steps SlOO and SllO of FIG. 1 a homogeneity criterion and a kind of 
matching are selected. However, in the next step S120 a type of complexity 
criterion to be minimized is selected. 

[000157] Such complexity criterion defines a defined value for the 
complexity of an entire structure consisting of all picture object and can be 
further defined as the sum through all picture objects in the hierarchical 
structure of standard deviations of color mean values of objects multiplied by 
the number of picture objects on a next lower hierarchical plane in each picture 
object as shown in equation (9). 



[000155] 



The method of the modification of the embodiment of the present 



c = 



* 

Udir ^Udir 



(9) 



i 



[000158] In equation (9) C designates the defined value for the complexity, 
^ iUdir designates the standard deviation of the color mean value of a 



respective picture object i, and niu^j^. designates the number of the picture 



objects on a next lower hierarchical plane in each picture object. 
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[000159] Furthermore, the complexity criterion can be further defined as 
the sum through all picture objects in the hierarchical structure of variances of 
color mean values of the picture objects on a next lower hierarchical plane in 
each picture object multiplied by the number of picture objects on the next 
lower hierarchical plane in each picture object as shown in equation (10). 



C = n. (10) 

var__ , . 1.. , . ^ ^ 

1 Udir Udir 



[000160] In. equation (10) C designates the defined value for the 
complexity, variy designates the variance of the color mean value of a 

respective picture object i, and niu^j^. designates the number of the picture 

objects on the next lower hierarchical plane in each picture object. 

[000161] Furthermore, in the same manner as described with respect to 

step 130 of FIG. 1 a picture object is selected. In the next step S440 a processing 

is determined which leads to a minimum increase of the complexity criterion. 

Such processing includes the operations aforementioned operations of 

merging, exclusion and boundary correction. This operations are shown in 

FIGs. 9 to 14. Due to the fact that steps of FIGs. 9 to 14 having the same 

reference signs as steps of FIGs. 3 to 6 are carried out in the same maimer the 

detailed description of these steps is omitted here. In addition, the operation 

"founding" can be carried out in such a manner that a new over-picture object 

is formed on the next higher hierarchical plane for two contiguous picture 

objects. This is shown in FIG. 15 in further detail. As is obvious from FIG. 15 

from to picture objects "1" and "2" a new picture object "1+2" is formed on the 

next hierarchy plane as an over-picture object of picture objects "1" and "2". It 

should be noted that respective modifications are performed on respective 
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highest local hierarchical planes of the hierarchical structure which is formed 
as described later. 



the next step S450. In the next step S460 it is determined in a similar manner as 
in step S170 whether a stable condition is reached. In the modification of the 
embodiment of the present invention such stable condition is reached if the 
entire picture is combined as one picture object on the highest hierarchical 
plane. Thus, the complexity criterion is a c9ontinuation criterion and not a 
termination criterion as in the embodiment of the present invention. 

[000163] In accordance with method the modification of the present 
invention no tolerance for the homogeneity criterion must be selected an a 
hierarchical structure is formed automatically by founding new picture objects 
on a higher hierarchical plane. This leads to a hierarchical structure having 
several hierarchical planes which are present in a locally different hierarchical 
depth as shown in FIG. 16. 

[000164] The following advantages can be achieved with the method of the 
present invention. 

[000165] The method of the present invention achieves high flexibility, 
high-quality segmentation in a desired resolution, simultaneous representation 
of the segmentation in various degrees of resolution and in the form of a 
hierarchical object structure, processing of highly textured pictures, separating 
highly textured picture regions from more homogeneous picture regions, 
separation of picture regions even when they present overlapping feature 



[000162] After it has been determined which processing leads to the 
minimum increase of the complexity criterion this processing is carried out in 
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ranges, processing of pictures having any dimensionality which are, for 
example, generated by a one-, two-, or three-dimensional imaging or other 
technique, and object-related admission of specific homogeneity criteria such 
that for example line structures or picture regions having a constant color 
development are also subjected to segmentation. 

[000166] The method of the present invention is distinguished from other 
segmentation methods, for example one- or multiple-threshold methods, by 
flexibility and quality. A particular strong point resides in the segmentation of 
textured pictures or data records, for example radar or X-ray pictures, 
something which often tums out to be difficult with other methods, in the 
concurrent segmentation of homogeneous and heterogeneous picture regions, 
in the simultaneous representation of several planes of resolution in the form 
of a hierarchical object structure, and in the concurrent representation of 
specific different picture structures, such as area and line segments or picture 
regions not having color development, and those having color development. 

[000167] The method largely works in the absence of beforehand 
Icnowledge, i.e. in the absence of beforehand knowledge with the exception of 
setting a tolerance, and is extremely flexible with respect to the desired picture 
resolution, i.e. the average number of picture elements per segment. The 
method can be performed largely in an automated manner, it does not require 
interactive intervention. It processes pictures or picture-type data records of 
any desired dimensionality. At a given resolution, very good segmentation 
results in terms of differential diagnosis are obtained. Identical structures are 
segmented jointly and separated from other structures. It is particularly also 
suited for finding and differentiating between textured picture objects having 
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overlapping feature ranges, or for processing high-noise pictures. Other than 
the numerous prior art methods, the method of the present invention does not 
employ pixel classification and subsequent segmentation. Picture elements are 
rather segmented based on homogeneity criteria in such a manner that picture 
structures are combined in an expedient manner in terms of differential 
diagnosis. With respect to quality, it is superior to segmentation methods 
relying on thresholds, and it is superior to methods which perform 
segmentation with respect to particular texture features, in flexibility of the 
desired resolution and the capability of processing most variegated textures at 
the same time. With the segments thus obtained, essentially more information 
is available for a subsequent classification than with single picture elements. 
• The classification probabilities thus made possible may therefore be increased 
distinctly. A maximum degree of condensation and utilisation of the existing 
information is achieved. This is not possible with prior art methods which 
employ partial ranges or distributions defined beforehand in the feature space 
and/or do not concurrently resolve in a plurality of planes. Differently 
textured objects are clearly delimited even when they present comparatively 
wide and overlapping feature distributions. 

[000168] In many cases the demands to a segmentation are such that entire 

pictures are to be expediently segmented into their components to then carry 
out an overall classification. This is in contrast with methods whereby single 
specific object types are to be determined. 

[000169] By the method described here an entire digital picture or a digital 

picture detail can be segmented entirely and very specifically with respect to 
detail even where little beforehand knowledge exists, which is also true if the 
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picture is at the same time highly textured or noisy. This segmentation is then 
very well suited as a basis for a semantic classification of the entire picture. 
Therefore the method can be employed for all types of digital pictures, 
independently from the picture generation method and of the dimensionality. 

[000170] The method is excellently suited for the processing of satellite 
and aerial pictures, of all kinds of medical images, namely of all two- 
dimensional as well as three-dimensional picture generation methods, but also 
for the simple picture object recognition. A particular suitability results for the 
segmentation and object recognition of three-dimensional pictures, such as for 
example in medical technology, which fact can be made use of for visualisation 
as well as for subsequent classification. 

[000171] Pertinent picture structures are segmented even without detailed 
beforehand knowledge about the picture and its structures, whereas different 
picture structures, even those having overlapping feature distributions, are 
separated from each other. The segmentation thus achieved represents a very 
much better result for a classification than a pixel classification, because very 
much more information is contained in whole segments than in single picture 
elements (feature distributions, texture, shape, size etc.), however also better 
than results of other segmentation methods, because the quality of the found 
segmentation is very good, and with the aid of the tolerance the resolution of 
the segmentation may very specifically be adapted to the requirements. This 
information helps to considerably improve certainty of classification. 

[000172] While a preferred embodiment of the present invention and a 

modification thereof has been shown and described, it will be understood that 

there is no intent to limit the present invention by such disclosure but rather it 
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is intended to cover all modifications falling within the spirit and scope of the 
present invention as defined in the appended claims. 
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